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FOREWORD 


The  Command  Systems  program  within  the  U.  S.  Army  Behavioral  Science  Research 
Laboratory  (BESRL)  is  concerned  with  problems  of  information  presentation,  processing, 
and  utilization  in  Command  an '  Control  Systems.  Specific  aspects  of  information  pro¬ 
cessing  systems  are  examined  with  respect  to  the  requirements  for  and  the  capabilities 
of  handling  information  and  its  presentation  to  the  human  users  of  the  system.  The  entire 
research  program  is  responsive  to  requirements  of  RDT&E  Project  2Q052106A723. 
“Human  Performance  in  Military  Systems,"  FY  1969  Work  Program. 

Coincident  with  BESRL  research  interests,  the  U.  S.  Army  Computer  Systems 
Command  (USAC.3C) ,  Headquarters,  United  States  Army,  Europe  (USAREUR).  and  the 
Seventh  Army  have  initiated  plans  for  the  design  verification  testing  of  an  automated 
Tactical  Operations  System  (TOS) .  This  test  program  is  aimed  at  verifying  the  efficacy 
of  the  design  of  a  field  system  conceptually  representative  of  the  operations  and  intel¬ 
ligence  aspects  of  the  Automatic  Data  Systems  within  the  Army  in  the  Field  (ADSAF) 
master  plan.  To  design  and  test  such  a  system,  *he  TOS  Development  Group  was  formed 
and  assigned  the  task  of  developing  a  field  version  of  the  10S  of  ADSAF.  The  TOS 
automates  aspects  of  Intelligence  (G2)  and  Operations  (G3J  functions  which  the  com¬ 
mander  may  monitor  and  use  to  direct  tactical  operations. 

In  September  1967,  the  Behavioral  Science  Research  Laboratory  established  a 
Command  Systems  Field  Branch  in  Europe  to  conduct  human  factors  performance  research 
in  connection  with  the  evaluation  of  ar.  experimental  Tactical  Operations  System.  In  the 
present  study,  problems  of  message  format  unique  to  the  experimental  TOS  were  examined 
and  a  newly  devised  job  aid  for  selecting  a  format  appropriate  to  a  given  message  was 
evaluated. 

BESRL  research  in  command  systems  is  conducted  as  an  in-house  research  effort 
augmented  by  research  contracts  with  organizations  selected  as  having  unique  capabili¬ 
ties  for  research  in  the  area.  The  present  study  was  conducted  jointly  by  personnel  of  the 
Behavioral  Science  Research  Laboratory  end  of  HRS-Singer,  incorporated,  under  program 
direction  of  Seymour  Hinge!. 


J.  E.  UHLANER,  Director  ^ 

U.  S.  Army  Behavioral  Science 
Research  Laboratory 


Requirement: 


The  experimental  version  of  the  automated  Tactical  Operations  System  (TOS) 
utilizes  over  40  different  message  formats.  At  the  present  stage  of  TOS  development,  G3 
staff  action  officers  have  to  determine  which  message  format  or  formats  to  use  with  each 
set  of  incoming  data.  The  present  study  was  conducted  by  BESRL's  Command  Systems 
F;e!d  Branch  to  examine  systematically  the  human  factors  problems  related  to  error  rate, 
processing  time,  and  confidence  in  format  selection.  A  second  objective  was  to  evaluate 
a  newly  devised  job  aid  for  use  in  selecting  the  appropriate  formats. 


Procedure: 

Forty-seven  simple  messages  were  given  to  14  individuals  familiar  with  TOS  or  G3 
operations,  or  both.  Their  task  was  to  select  an  appropriate  format  “for  each  message. 
Half  the  men  used  a  simple  job  aid  devised  by  the  experimenters,  the  other  half  used  a 
"menu"  type  listing  of  available  formats.  The  time  taken  to  complete  the  entire  task 
was  recorded.  Individuals  also  rated  the  degree  of  confidence  they  had  that  the  proper 
format  had  been  selected. 


Findings: 

Average  error  rate  in  format  selection  was  22%.  Mean  time  to  read  the  message  and 
select  a  format  was  approximately  50  seconds. 

Performance  with  the  job  aid  was  neither  better  nor  worse  than  with  the  menu  type 
listing. 

Messages  of  different  types  and  subject  matter  differed  in  error  rate  of  format 
selection. 


Utilization  of  Findings: 

The  study  provided  baseline  data  with  which  future  performance  can  be  compared. 
The  findings  suggested  some  approaches  to  training  and  alternative  methods  for  the 
transform  process. 
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THE  TRANSFORM  OPERATION  IN  TOS:  ASSESSMENT  OF  THE  HUMAN  COMPONENT 


SCOPE  OF  THE  STUDY 


Operational  Framework 

The  U.  S.  Army  is  developing  an  automated  Tactical  Operations  System 
(TOS)  to  assist  commanders  and  their  staffs  in  tne  conduct  of  tactical 
operations  by  collecting,  processing,  and  summarizing  information  re¬ 
quired  for  command  decisions  and  staff  actions.  The  development  is  part 
of  an  Army-wide  project,  the  Automatic  Data  Systems  within  the  Army  in 
the  Field  (ADSAF).  In  TOS,  emphasis  is  on  the  intelligence  (G?)  and 
Operations  (G3)  functions. 

While  the  introduction  of  automatic  data  processing  (ADP)  equipment 
into  tactical  operations  can  be  expected  to  speed  up  data  processing  and 
reduce  errors,  the  effective  performance  of  the  system  depends  ultimately 
on  its  human  component.  The  introduction  of  ADP  alone  cannot  be  expected 
to  extend  significantly  the  capabilities  of  the  system  nor  alleviate  the 
limitations  of  man  as  an  information  processor.  Further,  it  is  realistic 
to  expect  that  the  introduction  of  ADP  into  tactical  operations  will  pro¬ 
duce  new  human  performance  problems.  Solutions  to  these  problems  there¬ 
fore  become  of  paramount  importance.  In  recognition  of  this  fact,  BESRL 
has  established  the  Command  Systems  Field  Branch  to  work  closely  with  TOS 
developers  to  enhance  system  performance  through  facilitation  of  opera¬ 
tions  depending  wholly  or  in  part  on  the  performance  of  personnel  in  the 
system. 

In  laying  out  a  program  of  research  aimed  at  resolving  some  of  these 
problems,  Ringel1""  speculated  that  at  least  five  critical  human  perform¬ 
ance  areas  requiring  experimental  attention  would  accompany  the  advent  of 
the  TOS.  In  his  words: 

"An  automated  TOS  will  receive  vast  amounts  of 
information  from  many  and  varied  sources.  The 
information  will  vary  widely  in  content,  form, 
and  degree  of  completeness  .  .  .  The  raw  data 
will  require  a  great  deal  of  handling  and  pro¬ 
cessing  by  man  and  equipment  .  .  .  Looking  at 
the  system  as  a  whole,  there  appear  to  be  five 
critical  operations  that  man  and  equipment  will 
have  to  perform: 

1 .  Screen  incoming  data  for  pertinence,  cred¬ 
ibility,  Impact,  priority,  and  routing. 


Ringel,  S.  Command  information  processing  systems — A  human  factors 
research  program.  BESRL  Technical  Research  Report  1148,  June  I966. 


2.  Transform  the  raw  data  for  input  into  storage 
devices. 

3*  Input  the  transformed  data  into  storage  devices 
for  subsequent  computation  and  display. 

4.  Assimilate  data  displayed. 

5.  Decide  on  courses  of  action  based  on  informa¬ 
tion  displayed  and  information  from  other  sources." 

The  flow  of  these  operations  is  diagrammed  in  Figure  1.  While  each 
of  the  areas  cited  by  Ringel  has  special  significance  to  the  operation 
of  automated  systems,  each  of  these  operations  exists  also  in  completely 
manual  systems.  The  specifics  governing  methods  of  operation  may  change, 
but  the  overall  purpose  remains  the  same.  Screening  is  always  required 
to  aid  in  the  decision  about  what  information  should  enter  the  system. 

Data  transformation  is  necessary  to  organize  material  in  a  way  which  at 
least  partially  facilitates  storage  and  retrieval.  Although  the  popular 
connotation  of  the  word  "inputting"  suggests  automation,  a  broader  view 
of  the  term  could  include  making  entries  in  charts,  journal  files,  and 
other  reference  materials  in  manual  systems.  The  point  made  here  is  that 
at  least  some  of  the  problems  which  confronted  command  and  control  infor¬ 
mation  systems  before  automation  will  continue  to  be  problems  within 
automated  systems  as  well. 

These  basic  human  factor  problems  are  compounded  by  the  fact  that 
the  trend  in  the  design  of  military  information  systems  is  to  allow  the 
user  on-line  communication  with  the  computer  and  intervention  in  computer 
operations.  Accompanying  this  trend  has  been  the  development  of  rigidly 
defined  user/computer  languages.  The  languages  these  systems  use  are, 
and  within  the  current  state-of-the  art  must  be,  most  precise.  They  are 
intolerant  of  errors,  Errors,  even  if  slight,  usually  result  in  the 
computer's  rejecting  a  message. 

This  typical  inflexibility  of  computerized  information  systems 
accounts,  in  part,  for  the  first  two  human  performance  problem  areas 
which  Ringel  identified.  Screening  is  required  because  most  systems  are 
not  capable  of  accepting  every  iota  of  data.  Redundant,  irrelevant,  and 
unimportant  data  are  filtered  out  because  the  software  is  typically  un¬ 
prepared  to  respond  appropriately.  Trans  format ion  generally  involves 
formatting  and  translating.  "Formatting"  is  necessary  because  of  input 
hardware  restrictions  and  the  fact  that  computers  read  character  positions 
or  fields,  and  not  words  or  sentences.  A  further  restriction  is  that 
software  frequently  assigns  differential  meanings  to  positions  and  fields. 
Thus  a  "2"  in  column  one  is  not  necessarily  equal  to  a  ”2"  in  column  ten. 
This  restriction  leads  to  translation  requirements.  The  language  of  the 
"real  world"  must  be  converted  to  the  symbolism  of  the  computer.  In 
spite  of  the  fact  that  sophisticated  hardware  is  capable  of  converting 
alphanumerics  to  computer-compatible  electronic  binary  representations, 
the  software  "expects"  the  input  alphanumerics  to  conform  to  a  specified 
form.  Therefore,  certain  words  must  be  abbreviated  in  a  particular  way, 
special  names  converted  to  numerical  representations,  continuous  measure¬ 
ments  categorized  into  interval  scores,  exponential  notation  substituted 
to  shorten  number  length,  and  sundry  other  symbolic  representations 
implemented  to  synthesize  input  data. 


Thus,  the  macro- Jtructure  introduced  by  Ringel^ concerning  the  opera¬ 
tions  and  information  flow  for  an  automated  TOS  can  be  made  more  specific 
by  inserting  micro-structure  descriptions  into  the  "black  boxes"  he  pre¬ 
sented.  In  Figure  2,  the  insertions  have  been  made  for  Screen  and  Trans- 
form  operations.  The  micro-structure  presented  in  Figure  2  is  specific 
to  the  current  Seventh  Army  TOS  in  its  present  state  of  development.  The 
present  TOS  has  reached  a  stage  in  its  development  where  speculation  con¬ 
cerning  these  human  performance  areas  can  be  replaced  by  system- specific 
experimentally  generated  data. 

Concern  in  the  present  study  was  with  a  particular  aspect  of  the 
transform  process  which  is  peculiar  to  the  formatting  requirements  of 
the  Seventh  Army  TOS. 


Problems  of  Fo-mat  Selection 

Message  Worksheet  Approach .  Because  t'.ie  user / computer  language  c 
TG3  requires  exact  syntactic  structure,  abLrc*.  ifetions,  and  t  amino  lory  m 
order  for  the  message  to  be  accepted  by  the  computer.,  the  cyoicsDy  un¬ 
constrained  tree- English  text  that;  characterizes  Gp  message'  in  the  manual 
system  must  be  transformed  into  a  rigidly  defined  rormat  oefore  the  user 
can  interact  with  the  system.  This  operation  will  be  accomplished  in 
TOS,  in  accordance  with  current  plans,  by  having  a  03  staff  action  officer 
format  the  input,  utilizing  a  message  worksheet.  This  message  worksheet 
is  subseauently  used  to  proviae  the  text  for  the  User  Input-Output  Devi ce 
(UIOD)  operator  when  he  enters  the  message  into  the  system.  Additionally, 
the  staff  action  officer  will  interpret  any  resulting  output  messages 
arriving  at  his  UIOD.  These  messages  will  be  structured  essentially  the 
same  as  input  messages. 

To  illustrate  this  process,  assume  that  the  very  busy  G3  at  Seventh 
Army  Main  has  appointed  a  TOS  staff  action  officer  to  assist  him.  He 
hands  this  staff  action  officer  the  following  unclassified  message: 

I  WANT  TO  KNOW  IMMEDIATELY  ALL  U.  S.  COMBAT  ’JNL’S 
SUBORDINATE  TO  V  CORPS,  OF  DIVISION  ECHELON  OR 
LESS,  WHICH  HAVE  MORE  THAN  80  PERCENT  TOE  PERSON¬ 
NEL  AND  MORE  THAN  TO  PERCENT  COMBAT  EFFECTIVENESS. 
ADDITIONALLY,  THESE  UNITS  MUST  HAVE  AT  LEAST  fOO 
PERSONNEL. 


See  footnote  1  on  page  1. 


The  staff  action  officer  notes  that  the  Seventh  Array  G3  is  interested 
in  the  present  size  and  combat  capability  of  specific  units.  The  G3  mes¬ 
sage  requires  a  reply  which  ICS  ic  capable  of  providing  and  the  informa¬ 
tion  is  requested  on  a  one-time  basis.  The  Unit  Status  Query  message 
(see  worksheet.  Figure  3)  is  designed  specifically  to  meet  these  criteria. 
The  ataff  action  officer  begins  to  complete  it  once  the  proper  format  is 
selected »  His  job  is  to  transform  the  message  from  the  vernacular  into 
a  language  the  computer  understands  and  will  accept.-^ 


Complexity  o*  the  Format  Selection  Procedure 

In  the  situation  just  described,  an  "ideal"  staff  action  officer 
screened  incoming  messages.  He  had  no  difficulty  in  determining  that  the 
information  request  cited  here  was  pertinent  to  the  TOS  data  base. 

Further,  he  encountered  no  difficulty  in  selecting  the  correct  format 
worksheet  and  presumably  had  no  trouble  completing  it.  But  what  about 
She  "real  men"  who  will  be  using  the  system? 

The  present  study  vas  conducted  to  provide  seme  baseline  data  on  the 
transform  function.  The  primary  question  addressed  was  a  practical  one, 
namely:  Xu  the  present  version  of  the  experimental  TOS,  is  the  task  of 
selecting  an  appropriate  message  format  to  use  for  a  particular  message 
s  difficult  or  a  time- consuming  one?  If  so,  just  how  difficult  is  the 
task  and  what  error  rates  can  be  anticipated:  For  example,  listed  in 
Table  1  are  -13  message-worksheet  formats  specially  intended  for  use  by 
the  G3  element  i.n  the  TOC.  Cine  additional  message-worksheet  formats 
common  to  all  staff  elements  are  not  incl-uded  ir.  this  table  (one  excep¬ 
tion  in  the  Relay  message  whicn.  is  included  because  cf  ir.s  relation  to 
the  question  at  hand).  On  the  basis  of  number  alone,  this  basic  operation 
of  format  selection  is  potentially  troublesome  for  the  system. 

It  is  unlikely  rhat  even  the  ideal  staff  action  officer  would  commit 
to  memory  the  operator  communication  field  code  number  and  the  worksheet 
subject  titles  shown  in  Table  1,  although  after  a  while  he  may  remember  a 
few  key  ones.  More  likely,  in  normal  operations  he  would  refer  to  a  "menu 
or  index,  similar  to  Table  1,  when  presented  with  the  task  of  pairing  a 
message  with  an  appropriate  worksheet  or  worksheets.  But  a  "menu"  type 
job-aid,  although  typically  used  in  this  kind  of  situation,  may  not  be 
best  suited  to  the  task.  A  secondary  question  was  therefore  raised: 

Could  a  job-aic  be  developed  which  would  improve  human  performance  in 
format  selection? 


^The  reader  interested  in  pursuing  this  translation  process  through  al 
the  steps  to  completion,  i.e.,  to  the  point  where  the  worksheet  shown 
in  Figure  3  has  been  completed,  should  consult  Appendix  A. 
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Data  Input  Messages 

UA1®  Task  Organization  Data  Message 

UA2  Task  Organization  Data  change  Message 

TJA5  Task  Organization  Data  Delete  Message 

UB1  Task  Force  Data  Message 

UB2  Task  Force  Data  Change  Message 

UB3  Task  Force  Data  Delete  Message 

UC2  FLT  and  Coordinating  Points  Data  Change  Message 

UD2  Command  Post  and  Center  of  Mass  Data  Change  Message 

DE2  Boundary  Data  Change  Message 

UF2  Control  Lines  Data  Change  Message 

DG2  Current  Activity  Data  Change  Message 

UH2  Planned  Activity  Data  Change  Message 

UJ2  Unit  Status  Data  Change  Message 

UL2  SITREP  Data  Change  Message 

UMJ  Periodic  Preparation  of  SITREP  Delete  Message 
UN3  Pending  Change  Delete  Message 


UA4  Task  Organization/Task  Force  Query  Message 

UD4  Conmand  Post,  Center  of  Mass  Locations  Query  Message 

UE4  Boundaries  Query  Message 

UF4  Control  Lines  Query  Message 

UG4  Operational  Activity  Query  Message 

UJ4  Unit  Status  Query  Message 

UL4  Situation  Report  Query  Message 

UN4  Pending  Change  Query  Message 

SRI  Messages 

UA6  Task  Organization/Task  Force  SRI  Establish  Message 

UA7  Task  Organization/Task  Force  SRI  Change  Message 

UB6  CP/CCM  Locations  SRI  Establish  Message 

UD7  CP/CCM  Locations  SRI  Change  Message 

UE6  Boundary  SRI  Establish  Message 

UE7  Boundary  SRI  Change  Message 

UF6  Control  Lines  SRI  Establish  Message 

UI7  Control  Lines  SRI  Change  Message 

UG6  Operational  Activity  SRI  Establish  Message 

UG7  Operational  Activity  SRI  Change  Message 

UJ6  Unit  Status  SRI  Establish  Message 

UJ7  Unit  Status  SRI  Change  Message 

UX6  Front  Line  Trace  SRI  Establish  Message 

UK7  Front  Line  Trace  SRI  Change  Message 

UL6  SITREP  SRI  Establish  Message 

UL7  SITREP  SRI  Change  Message 

Special  Process  Requests 

UR5  Front  Line  Trace  Special  Process  Request  Message 
UL5  SITREP  Special  Process  Request 
UN5  Request  for  Periodic  SITREP 


Relay  Message 


AAO  Free  Text  Relay  Message 


“Operator  ccxrmunicstions  field  code  number  for  each  message. 
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METHOD 


Test  Material 

It  was  considered  desirable  to  keep  the  initial  study  fairly  simple. 
For  this  reason,  appropriate  message/situation  descriptions  were  abstracted 
from  existing  documentation  where  they  function  as  more  or  less  homogeneous 
examples  for  the  application  of  specific  formats. 

One  such  message  was  found  for  most  of  tne  format -worksheets  listed 
in  Table  1;  in  a  few  cases,  more  than  one  message  was  obtained.  The  re¬ 
sult  was  a  pool  of  G3  message/situation  descriptions  which  were  similar 
to  the  one  used  in  the  illustration  on  page  4  of  this  report. 


Sample 

A  list  was  compiled  of  military  and  civilian  personnel  working  on 
the  TOS  development.  All  were  considered  by  the  authors  to  be  experienced 
in  TOS  system  operations  or  in  G3  operations  or  both.  These  personnel 
were  contacted  and  asked  if  they  would  serve  as  subjects.  When  a  pool  of 
fourteen  was  reached,  the  men  were  roughly  matched  and  paired  in  terms  of 
length  of  time  they  had  worked  on  TOS  development  and  experience  in  and 
related  knowledge  of  G3  operations.  This  procedure  provided  two  groups 
of  equated  subjects,  one  "standard"  and  the  other  "job-aided." 


Testing  Procedure 

The  instructions  presented  to  both  groups  are  reproduced  in  Appendix  B 
Essentially,  the  standard  group  was  told  that  they  would  be  given  a  set  of 
message/situation  descriptions  which  they  were  to  read.  They  were  then  to 
select  from  a  list  of  message  formats  (Table  1)  the  one  which  was  most 
appropriate  for  the  situation  covered  in  each  description.  They  were  also 
informed  that  each  situation  represented  an  isolated  and  independent  case, 
and  that  it  should  not  be  considered  in  relation  to  other  situations. 

The  other  group  was  treated  exactly  the  same  way,  with  the  exception 
that  they  were  given  a  job-aid  (Figure  4)  to  assist  them  in  finding  the 
correct  format  instead  of  the  Table  1  list  used  by  the  standard  group. 

All  subjects  progressed  through  the  test  booklet  of  message/situation 
descriptions  at  their  own  pace.  When  they  had  selected  what  they  con¬ 
sidered  was  the  appropriate  message  format,  they  entered  the  corresponding 
alpha-numeric  operator  communic ttions  field  code  in  the  space  provided 
(see  Figure  5)«  Each  subject  ‘‘hen  checked  the  block  at  the  bottom  of  the 
response  sheet  which  be3t  expressed  his  certainty  concerning  the  correct¬ 
ness  of  his  selection  (Figure  5)»  When  he  had  completed  tho  task,  he 
noted  his  finish  time  on  the  test  booklet  and  then  turned  it  in  to  the 
experimenter . 


Example:  The  Third  Battalion,  Seventh- sixth  Armor,  is  moving 

up  causing  a  change  to  its  boundaries.  Spearhead  TOC 
sends  the  following  unclassified  message:  "3-BN-76-ARMD" 
moving  up  to  on-line  position  2.  Left  boundary  LA31927864 
and  LA30647672.  Right  boundary  LA33927461  and  LA32847367 • 
Effective  time  of  boundaries  is  15$4$$ZAUG67 • 

Message  Format  U  £  H- 


My  certainty  of  the  message  format  being  correct  is: 


Absolutely 

Uncertain 

Very 

Uncertain 

Moderately 

Uncertain 

l 

SUWly 

Unperb^in 

Moderately 

Certain 

Very 

Certain 

Absolutely 

Certain 

_ ! 

Figure  5.  Sample  massage/situation  description  used  in  the  experiment 

Subject  has  responded  that  the  message  format  needed  to  enter  these  data  into  TOS  is  a  UE2. 
i.e  '..  a  "Boundary  Data  Change  Message."  He  has  also  indicated  that  he  was  "slightly  un¬ 
certain"  when  he  made  this  response.  The  certitude  scale  used  in  this  study  was  developed 
hry  R.  Andrews  and  S.  Ringel  I  Andrews.  R.  S.  and  S-  Ringel.  Certitude  judgments  and  the 
accuracy  of  information  assimilation  in  visual  displays.  BESRL  Technical  Research  Note  145. 
April  1554:1 


RESULTS 

Job-Aid  Evaluation^ 

One  of  the  questions  posed  in  this  study  was:  Could  a  job-aid  be 
developed,  specific  to  this  task,  which  would  improve  human  performance? 
The  proposed  aid  is  illustrated  in  Figure  4.  Results  summarized  in 
Table  2  indicated  that  the  answer  is  no,  at  least,  not  within  the  limits 


^Although  this  aspect  of  the  study  was  considered  a  secondary  objective, 
the  analyses  of  these  data  are  presented  first.  The  reason  for  this 
ordering  will  become  obvious  when  the  subsequent  data  analyses  which 
support  uhe  rationale  for  pooling  the  data  are  presented. 
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of  the  present  study A'  Upon  examination,  the  data  did  not  warrant 
further  analysis,  since  it  was  evident  that  no  practical  differences 
existed. 


Table  2 

COMPARISON  SCORES  OF  THE  EQUATED  SUBJECTS  (Ss)  ON  THE  THREE  MEASURES 
OBTAINED  IN  THE  STUDY— ERRORS,  TIME  TO  COMPLETION, 

AND  EXPRESSED  CERTITUDE 


..  - - , 

Pair 

I  Errors 

Time  ( in 

minutes) 

Average 

Certitude 

(Job- aid 
Group) 

(Standard 

Group) 

(Job-aid 

Group) 

(Standard 

Group) 

(Job- aid 
Group) 

( Standard 
Group) 

A 

5 

2  • 

49 

30 

6.7 

7.6 

B 

4 

9 

60 

37 

5-2 

5.0 

C 

9 

8 

32 

34 

.  7-5 

5-7 

D 

7 

7 

31 

55 

5-4 

6.0 

E 

10 

5 

36 

26 

6.5 

7-2 

F 

15 

17 

30 

60 

4.3 

3-9 

G 

16 

21 

46 

39 

3*2 

4.5 

Average 

9-43 

9-86 

40.6 

57.3 

5-5 

5-7 

^One  of  the  limits  of  the  study  was  that  the  standard  and  job-aided 
groups  could  only  be  assumed  to  be  equivalent.  No  independent  com¬ 
parative  measure  of  the  efficacy  of  this  pairing  was  obtained.  Time 
factors,  not  to  be  delineated  here,  precluded  the  development  of  an 
appropriate  device  for  obtaining  the  required  measures.  Use  of  each  £ 
as  his  own  control  (i.e.,  each  J3  serving  under  both  conditions  in  some 
counter-balanced  order)  was  eliminated  as  a  means  for  getting  around 
this  problem.  The  reason  was  twofold,  one  pragmatic  and  the  other  a 
study  design  problem.  The  pragmatic  reason  is  simply  that  the  limited 
number  of  appropriate  stimulus  materials  ruled  out  this  approach.  In 
terms  of  experimental  design,  serial  learning  effects  encountered  in 
this  type  of  experimental  task  are  very  difficult  to  handle  statisti¬ 
cally.  Thus,  even  if  paucity  of  appropriate  stimuli  could  have  been 
overcome,  another  negative  factor  would  have  been  encountered. 
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Accuracy,  Time,  and  Certitude 

The  time  and  certitude  data  (Table  2)  were  btained  using  test 
booklets  which  contained  47  situation/messages  'Figure  6).  The  time  data 
were  obtained  by  measuring  from  start  time  of  session  to  finish  time  and 
were  not  item- by- item  measures.  Hence,  "average  time  per  item"  statements 
are  based  tipo-.  computations  for  47  items.  However,  the  arror  score  data 

and  the  av<r  age  certainty  data  are  based  on  an  N  of  44  items,  a  drop  of 

three  items  from  the  total  number.  First,  two  situation  messages  were 
inadvertently  replicated  when  the  booklets  were  collated  and  the  two 
extra  items,  were  dropped  from  the  analysis.  Second,  during  data  analysis, 
one  item  was  found  that  every  subject  answered  incorrectly.  A  careful 
examination  of  tne  item  indicated  that  no  unequivocal  criteri >n  existed 
for  stating  that  any  available  response  was  in  fact  correct.  Hence,  it 

was  assumed  to  be  a  "bad"  item  and  dropped  from  the  sample. 

The  data  summarized  in  Table  2  show  that  the  two  groups  were  nearly 
identical  in  terms  of  mean  number  of  errors  made  (job-aided, 
standard,  9 -36).  The  average  time  required  to  complete  the  task  was 
essentially  the  same  for  both  groups  (job-aided,  40.6  minutes;  standard, 
37-3  minutes).  The  mean  expressed  certainty  that  the  selection  of  format 
worksheet  was  in  fact  correct  was  almost  the  same  (job-aided,  5*5>  stan¬ 
dard,  5-7) •  This  particular  finding  came  as  a  surprise  since  it  was 
originally  hypothesized  that  the  men  using  the  aid  would,  at  the  very 
least,  be  more  confident  in  their  choice.  On  the  basis  of  these  data, 
it  must  be  concluded  that  the  job-aid  failed  to  improve  performance  in 
the  present  study. 


Overall  Performance 

Since,  as  noted  in  the  job-aid  evaluation  section,  no  significant 
differences  between  the  two  groups  were  found,  the  data  from  the  two 
groups  were  combined  for  purposes  of  the  overall  analysis. 

In  terms  of  errors,  the  average  rate  for  the  total  sample  was  9-6. 
Stated  another  way,  an  average  of  9.6  errors  in  the  experimental  situa¬ 
tion  involving  44  messages  represents  a  22  percent  error  rate.  The 
reader  should  keej  in  mind  that  the  messages  used  in  this  experiment 
were  purposely  kep :  simple  and  that  the  experimenters  were  in  effect 
begging  the  question  with  regard  to  level  of  "user"  performance  in  the 
case  of  the  more  complex  task  of  determining  how  many  and  what  kinds  of 
formats  are  required  for  messages  that  cannot  be  accommodated  by  a 
single  format. 

By  the  some  token,  these  "users"  were  going  into  the  situation 
"cold."  With  experience,  error  rate  could  be  expected  to  decrease.  For 
example,  the  feedback-loop  shown  in  Figure  2  (seeing  whether  the  format 
selected  in  fact  acconmodates  the  data  in  the  message)  should  temper  sub¬ 
sequent  selections.  While  this  assumption  appears  logically  sound,  its 
validity  remains  to  be  tested. 
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Mean  format  selection  time  was  49.8  seconds  per  message.  Interviews 
with  G3  action  officers  and  the  analysis  of  data  obtained  during  Corps 
and  Division  exercises  have  revealed  that  the  G3  action  officer,  during 
peak  periods,  may  expect  to  receive  an  average  of  40  to  50  messages  per 
hour In  other  words,  the  action  officer  could  be  dealing  with  a  new 
message  every  70  to  90  seconds.  Rowever,  not  all  these  messages  will  be 
applicable  to  the  TOS.  In  any  case,  the  time  required  simply  to  choose 
an  appropriate  message  format  for  messages  applicable  to  TOS  is  nearly 
50  seconds.  It  seems  reasonable  to  assume  that  the  G3  action  officer 
could  be  quickly  overloaded  in  peak  action  periods. 

Even  if  these  time  comparisons  could  be  disregarded,  there  are 
several  other  reasons  to  expect  that,  unless  precautions  are  taken,  the 
action  officer  will  become  over.  ided.  At  least  some  of  the  messages  he 
will  receive  during  peak  periods  are  likely  to  be  more  complex  than  those 
included  in  the  stimulus  set  of  the  present  experiment,  perhaps  requiring 
multiple  formats  for  inserting  all  the  information  into  the  data  base. 
Consequently,  both  error  rate  and  response  time  are  likely  to  increase. 
Additionally,  some  portion  of  an  action  officer's  time  will  be  spent 
screening  the  incoming  data  to  determine  the  appropriateness  to  TOS  (see 
Figure  2).  The  data  then  will  need  to  be  translated  into  computer- 
acceptable  TOS  language.  As  noted  earlier,  some  translation  effort  is 
also  involved  in  handling  TOS  output.  Finally,  the  action  officer  will 
be  involved  in  correcting  errors  which  are  bound  to  occur  at  each  step 
during  this  information  handling. 

Taken  together,  the  error  rate  and  time  data  just  presented,  as  well 
as  the  impact  of  the  other  steps  described,  lead  to  the  conclusion  that 
the  transformation  process  is  potentially  a  major  problem  in  the  experi¬ 
mental  TOS. 


ANALYSIS  OF  ERROR  SOURCE 

A  detailed  examination  of  the  messages  which  produced  the  most  fre¬ 
quent  errors  was  then  undertaken.  The  intention  was  to  determine  if  the 
pattern  of  errors  could  provide  some  meaningful  insights  into  the  "why" 
behind  the  subjectively  high  error  rate  observed  in  this  study. 


The  interviews,  supplemented  by  a  questionnaire,  were  conducted  by  the 
authors  in  conjunction  with  a  corps  level  exercise.  The  manual  system 
data  were  extracted  from  a  report  by  Reese  C.  Wilson  of  the  User  Plan¬ 
ning  Group  entitled:  ''FRONT  CENTRE  68  -  Data  Reduction  and  Analysis 
Results,"  dated  May  I968. 


The  percentage  error  rates  presented  in  Table  3  were  computed  for 
individual  message  subjects  and  message  types.  The  percentages  were 
weighted  to  take  into  account  the  differential  frequency  of  occurrence 
of  each  message  subject  and.  type  in  the  total  sample.  There  were,  for 
example,  six  messages  in  the  stimulus  set  with  "Task.  Organization"  as 
the  subject.  These  six  messages  were  vated  by  the  14  subjects.  They 
made  17  errors — a  weighted  error  rate  of  20  percent  (S  x  14  -  84; 

17  -r  84  »  20$).  Table  3  reveals  that  certain  of  the  message  subjects 
and  message  types  contributed  more  than  others  to  the  error  rate. 

Message  subjects  concerned  with  "Planned  Activity"  and  message  types  of 
the  "Data  Add"  variety  were  clearly  front-runners  in  the  production  of 
erroneous  responses.  Table  3  also  shows  that  the  difference  between  the 
message  subject  errors  and  message  type  errors  was  minimal  (95  vs  89 
respectively) . 


Analysis  of  Erroneous  Response  Patterns 

While  the  data  in  Table  3  serve  to  point  out  which  message  subjects 
and  message  types  contributed  most  to  errors,  they  do  not  reveal  the 
nature  of  the  confusion.  To  identify  possible  reasons  for  confusion  in 
the  responses  made,  a  detailed  examination  was  made  of  a  selected  set  of 
the  stimulus  items  in  the  sample  and  subject  responses.  The  items 
selected  were  those  which  produced  error  rates  above  the  median  of  the 
total  distribution  of  weighted  error  scores.  Results  of  this  examination 
are  summarized  In  Table  4. 

In  Table  4,  the  massage  categories  have  been  rank-ordered  In  terms 
of  highest  to  lowest  error  score.  Associates  with  each  category  it*  tne 
most  frequent  erroneous  response  to  that  message  category.  Also  shorn  -.3 
the  likelihood,  given  an  error,  that  the  error  will  be  cf.  a  particular 
type.  For  example,  if  the  stimulus  message  was  one  which  or 5 gina tod  free 
a  situation  requiring  a  "Planned  Activity"  format,  the  probability  of  an 
error's  occurring  on  that  item  was  .5.  Stated  another  way,  50$  of  the 
responses  made  to  these  messages  were  in  error.  Cf  the  errors  made,  57$ 
consisted  of  responding  that  the  data  called  for  a  "Relay  Massage”  re¬ 
sponse,  while  the  remaining  43$  consisted  of  erroneously  stating  that  a 
"Task  Force"  format  was  required. 

To  present  the  data  in  Table  4  in  the  frfimework  of  an  operational 
analogy,  if  the  user  in  the  field  performs  as  the  subjects  in  th-  study 
did,  he  will  be  serving  as  a  buffer  in  the  transform  process,  depriving 
the  data  base  of  valid  information  (if  he  selects  a  relay  message)  and 
introducing  time  delays  in  the  data  base  up-date  (if  he  selects  an 
erroneous  format  in  an  attempt  tc  "wedge"  the  data  into  the  system). 

The  impact  of  the  man  in  the  system  is  illustrated  in  Figure  7,  by  taking 
some  liberties  with  the  data  in  Table  4. 

In  spite  of  the  small  number  of  messages  involved,  the  data  in 
Table  4  are  nonetheless  suggestive  of  some  significant  sources  of  error 
in  the  transform  process. 
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PERCENT  ERRORS  FOR  MESSAGE  SUBJECT  AND  MESSAGE  TYFE  CATEGORIES 


Cod^ 

Message  Categories 

Number 

in 

Sample 

Total  Number 
of 

Errors 

Percent 

(Weighted) 

Errors 

Subject 

A 

Task  Organization 

6 

17 

20$ 

B 

Task  Force 

3 

14 

35$  + 

FLT  and  Coord,  Points 

0 

0 

— 

D 

CF  and  CUM  Locations 

5 

8 

n$ 

E 

Boundary 

4 

5 

9$ 

F 

Control  Lines 

A 

5 

9$ 

G 

Current  Activity 

4 

9 

16$  + 

H 

Planned  Activity 

1 

7 

50$  + 

J 

Unit  Status 

5 

8 

11$ 

K 

Front  Line  Trace 

2 

4 

14$ 

L 

SITREP 

s 

8 

11$ 

M 

Periodic  Prep,  at  SITREP 

2 

6 

21$  + 

N 

Pending  Change 

3 

10$ 

A 

Relay 

0 

0 

*  “  — 

Sub-Total 

44 

93 

Im. 

1 

Data  xivxc 

2 

7 

2%  + 

R 

Data  Chany? 

9 

22 

17$  + 

3 

Data  Delete 

4 

6 

11$ 

4 

Query 

11 

14 

9$ 

c. 

Special  Process 

3 

10 

24$  + 

6 

SRI  Establish 

8 

26 

23$  + 

T 

SRI  Change 

7 

4 

4$ 

0 

Relay 

0 

0 

Sub-Total 

44 

89 

aCo=j  as  used  it.  -.his  and  the  tabiss  to  follow  is  fan  abbreviate-*!  form  of  the  Operator  Communication  Fiald  Coda  given  trs 
Table  2.  For  sxerrcle.  UA1  ,s  *he  coO«  foi  o  task  o.’gan.aation  rata  message  in  Table  1  while  in  this  table  A  alone  is 
use*  to  designate  task  organiiatioa. 

bpij,  signs  r~signats  those  message  category  responsas  with  especially  high  error  rates. 
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Figure  7.  Illustrates  a  hypothetical  sequence  of  ten  "Planned  Activity"  messages  being 
processed  by  a  G3  staff  action  officer 

Of  these  ten  messages,  and  based  upon  the  data  in  Tabie  4.  we  could  expect  about 
half  of  the  messages  to  be  handled  erroneously.  Of  those  in  error,  approximately 
three— for  example,  messages  No.  2.  5.  and  6—  would  be  sent  out  as  relay  messages. 

The  staff  action  officer  also  would  have  attempted  to  compose  two  of  these  messages- 
e.g..  3  and  7— using  a  "Task  Force"  format  worksheet. 
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Confusion  of  Message  Content  and  Format  Label 

Several  years  ago,  Chapanis2-'  published  &  paper  in  which  he  called 
attention  to  a  very  large  and  important  area  of  human  factors  which  was 
then — and  for  that  matter  still  is — almost  entirely  neglected.  This  area 
consists  of  the  language  and  the  words  attached  to  the  tools,  machines, 
systems,  and  operations  with  which  human  factors  personnel  are  concerned. 
Examples,  illustrations,  ana  data  were  cited  to  show  that  changes  in  the 
words  used  in  man-machine  systems  may  produce  greater  improvement  in  per¬ 
formance  than  do  changes  in  the  machine  itself. 

Evidence  in  the  present  data  suggests  that  the  human  performance 
problems  encountered  in  this  TOS-tied  transform  process  is  yet  one  more 
instance  of  the.  difficulties  individuals  have  with  the  words  and  language 
of  man-machine  systems.  The  problem  uncovered  here  is  one  not  generally 
encountered  in  laboratory  research  where  stimulus  materials  are  permitted 
to  vary  only  with  respect  to  well  defined  dimensions.  Although  in  the 
current  study  the  stimulus  materials  (messages)  were  said  to  vary  with 
respect  to  the  appropriate  format  category,  it  cannot  be  assumed  that  the 
resulting  nominal  scale  (message  format  titles)  reflects  corresponding 
variability  in  the  latent  content  of  the  messages  themselves.  The  format 
categories  and  message  content  are  never  isomorphic;  that  is,  while  the 
format  categories  refer  to  well  defined  groups  of  items,  the  content 
often  extends  beyond  the  purpose  implied  Sy  the  title. 

The  problem  of  format  selection  is  one  of  recognizing  the  correspon¬ 
dence  between  the  significant  characteristics  of  a  message  and  the  purpose 
of  one  of  the  message  formats.  Since  the  purpose  of  each  format  is  more 
complex  than  its  title  would  indicate,  attempting  to. match  key  items  in 
the  message  content  with  format  title  alone  is  not  sufficient.  Yet  this 
strategy  appears  to  be  the  one  used  by  the  subjects  in  many  instances,  as 
illustrated  by  comparing  the  content  of  selected  stimulus  messages  with 
the  types  of  error  reflected  in  Table  4.  The  only  "Planned  Activity" 
message  in  the  stimulus  set  was  as  follows: 

A  TASK  FORCE  NAMED  TF-TEST  IS  TO  MINE  APPROACHES 
•  1  BRIDGE,  THEN  MOVE  FORWARD  TO  SOUTH  OF  AREA 
NEAR  MISSION  X.  START  TIME  OF  THE  OPERATION  IS 
dOtd  ZULU  ON  16  AUGUST  I967,  AND  END  TIME  IS  <J8 M 
17  AUGUST  I967.  THE  UNIT  SENDING  THIS  UNCLASSIFIED 
MESSAGE  IS  THE  THIRD  ARMORED  DIVISION  AT  TOC. 


^Chapanis,  A.  Words,  words,  words.  Human  Factors,  I965,  1*  1-17  • 


The  message  content  suggests  that  the  simplest  explanation  for  con¬ 
fusion  of  the  "Planned  Activity"  massage  with  the  "Task  Force"  response 
is  that  the  single  Planned  Activity  message  dealt  with  the  planned  activ¬ 
ity  of  a  task  force.  The  opening  statement  is  "A  TASK  FORCE  NAMED  TF-TEST 
IS  TO  . ..,  etc."  This  explanation  gains  further  support  by  examining  the 
errors  for  the  four  Current  Activity  messages,  none  of  which  contained 
the  term  "Task  Force."  None  of  the  errors  for  these  messages  resulted  in 
a  Task  Force  response. 

The  errors  for  the  three  Task  Force  messages  can  be  explained  simi¬ 
larly.  Three  of  the  four  erroneous  "Task  Organization"  responses  were  to 
the  only  message  which  had  the  word  "organize"  as  part  of  its  content. 
Likewise,  three  of  the  four  erroneous  "Current  Activity"  responses  occurred 
to  the  only  message  which  specifically  discussed  an  activity,  a  river 
crossing. 

These  errors,  and  others  like  them,  indicate  that  subjects  were  often 
misled  by  an  irrelevant  correspondence  between  message  content  and  message 
category  titles.  Since  the  titles  are  not  completely  descriptive  of  the 
purpose  of  various  message  formats,  additional  familiarity  with  the  purpose 
of  each  format  through  a  carefully  structured  training  program  is  required 
if  errors  are  to  be  reduced.  Alternatively,  a  set  of  mnemonic  descrip¬ 
tions  for  message  titles  which  draws  attention  to  the  relevant  differences 
in  the  purpose  of  the  various  message  formats  could  act  to  reduce  error. 
Such  an  approach  would  require  further  experimentation  to  develop  discrim- 
inably  different  titles  associated  with  a  low  error  rate. 


Type-Subject  vs  Subject-Type  Procedure 

As  just  mentioned,  a  carefully  structured  training  program  could  con¬ 
ceivably  reduce  the  number  of  errors  in  the  field.  The  present  results 
yield  some  suggestions  regarding  this  training.  The  suggestions  were 
reinforced  by  statements  made  by  some  of  the  subjects  in  the  study. 

Post-experimental  discussions  with  the  subjects  suggested  that  the 
process  of  choosing  a  format  began  with  the  choice  of  a  message  type, 
followed  by  the  choice  of  a  message  subject  from  those  available  for  the 
type  of  message  chosen.  To  determine  how  this  "type-subject  hypothesis" 
describes  the  data,  expected  values  were  computed  for  each  response  format 
listed  in  Table  1  or  Figure  4,  and  these  were  correlated  with  the  observed 
error  rate  for  each  response. 

The  expected  values  were  calculated  under  the  assumption  that  erro¬ 
neous  responses  were  made  at  random  within  the  constraints  of  the  model; 
that  is,  the  total  sample  of  error  response  was  distributed  evenly  among 
the  8  message  types  and  the  errors  of  each  type  were  distributed  evenly 
among  the  various  message  subjects  within  the  type.  For  example,  the 
total  number  of  errors  was  1??,  an  expected  *Talue  of  16. 5  for  each 


message  type.2-'  Since  there  were  eight  subjects  associated  with  the  query 
type  message,  the  expected  error  value  of  each  (e.g.,  UA4,  UD4  etc,) 

would  be  2.06  M  .  Likewise,  since  the  Data-Add  message  type  had  only 

two  message  subjects,  the  expected  error  value  for  erroneous  UA1  and  UBi 
format  responses  would  be  8.25. 


The  correlation  between  expected  and  observed  errors  using  the  type- 
subject  hypothesis  was  r  =  .86.  To  determine  whether  the  hypothesis  was 
in  fact  better  than  its  converse  (the  subject-type  hypothesis),  a  corre¬ 
lation  coefficient  was  similarly  comput  d  on  the  assumption  that  subjects 
chose  message  subjects  first  and  then  chose  a  message  type  from  those 
available  for  that  subject.2-'  Expected  values  for  message  subjects  were 


therefore  9  *43 


m. 


Since  there  were  4  types  of  message  available  for 


the  message  subject,  "Control  Lines,"  the  format  responses  of  UF2,  UF4, 
UF6,  and  UFf  all  have  expected  values  2.41 


The  correlation  coefficient  obtained  for  the  subject-type  hypothesis 
was  .62,  which  was  less  than  the  coefficient  of  .86  obtained  using  the 
type-subject  hypothesis. 


The  implication  of  these  data  for  structuring  a  training  program  is 
straight-forward :  Emphasis  should  be  on  teaching  the  potential  user  the 
unique  aspects,  as  well  as  properties  and  characteristics  of,  the  type  of 
message  transaction  required,  and  only  limited  coverage  need  be  given  the 
message  subject  characteristics.  The  basis  for  this  conclusion  is  simply 
that  a  "user"  in  the  field,  when  faced  with  the  task  of  choosing  a  format, 
will  most  likely  select  a  message  type  first,  and  then  the  message  subject. 


^■'The  total  of  132  errors  represents  the  total  number  of  incorrect  re¬ 
sponses  made  by  Ss.  This  number  is  somewhat  less  than  the  sum  of 
message  subjects  (95)  ana  message  type  (89)  errors  because  several  of 
these  errors  occurred  as  joint  events,  i.e.,  the  subject  responded 
with  both  the  incorrect  type  and  incorrect  subject  to  the  same  message. 

^The  critical  reader  may  be  puzzled  at  this  point,  noticing  that  some 
message  formats  serve  purposes  of  two  message  subjects.  This  is  true 
of  formats  UA4,  UA6,  UA7,  UG4,  UG6  and  UG7 •  The  UG4  format,  for  ex¬ 
ample,  is  a  query  message  for  both  current  and  planned  activity.  Com¬ 
putation  of  expected  values  for  the  subject-type  hypothesis  considered 
these  dual  roles  by  first  postulating  the  existence  of  a  separate  format 
for  the  second  subject  of  each  of  these,  i.e.,  UB4,  UB6,  UB7,  UH4,  UH6 
and  TJH7.  Expected  values  were  calculated  as  described  in  the  text,  but 
since  the  identical  response  was  required  for  current  or  planned 
activity  and  Task  Organization  and  Task  Force  Query  and  SRI  messages, 
the  expected  values  of  T1A4  and  UB4,  UA6  and  UA7,  etc.,  were  summed  to 
produce  one  expected  value  for  each  format  in  the  response  set  which 
served  a  dual  role. 
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Relay  Message  Error 

One  other  type  of  error  merits  comment.  The  data  in  Table  4  shew 
that,  overall,  the  most  frequent  erroneous  response  was  in  selecting  a 
"Relay11  message  format.  The  relay  message  is  used  to  communicate  plain 
text  remarks — statements  in  normal  English  language — between  the  staff 
users  of  the  TOS.  Mhen  used,  the  message  performs  a  communication  func¬ 
tion  only:  it  has  no  effect  on  the  TOS  data  base  and  it  does  not  inter¬ 
face  with  other  TOS  messages.  It  is,  in  effect,  a  catch-all  category  and 
was  included  in  our  response  set  for  that  very  reason.  Consequently, 
there  was  no  item  in  the  stimulus  set  specifically  requiring  a  Relay 
Message  format  (see  Figure  6).  Many  of  the  personnel  involved  in  the 
TOS  development  (including  a  number  of  the  subjects  in  the  present  experi¬ 
ment)  have  speculated  that  the  relay  message  will  be  "overworked"  in  TOS. 
The  present  data  tend  to  bear  them  out. 


CONCLUSIONS 

The  primary  objective  of  the  study  was  to  produce  normative  data 
concerning  user  performance  levels.  Overall,  the  users  in  the  experiment 
operated  at  about  a  22  percent  error  rate.  On  the  average,  it  took  them 
about  50  seconds  per  message  to  select  a  format.  Certain  message- format 
combinations  contributed  more  to  user  confusion  than  did  others;  for  ex¬ 
ample,  a  situation  requiring  a  Planned  Activity  format  resulted  in  an 
error  about  50$  of  the  time,  while  situations  concerned  with  "Boundary" 
or  "Control  Line"  formats  resulted  in  few  errors.  A  detailed  analysis 
of  message  content  revealed  that  individuals  are  often  misled  by  irrele¬ 
vant  correspondence  between  message  content  and  message  category  titles, 
since  the  titles  are  not  completely  descriptive  of  the  purpose  of  the 
various  message  formats.  It  also  was  found  that  the  selection  of  the 
Relay  Message  format  (»  plain  text  communication  which  utilizes  TOS 
equipment  but  does  not  interact  with  the  TOS  data  base)  was  the  most 
frequent  erroneous  response  made  over  all  the  message  categories. 

The  results  point  out  a  number  of  considerations.  First,  the  error 
results.  Even  in  situations  where  messages  are  simple  and  require  only 
a  single  format,  a  relatively  high  error  rate  in  format  selection  can  be 
anticipated.  This  anticipated  high  error  rate  must  be  qualified,  since 
the  subjects  in  the  present  study  were  not  formally  trained  in  format 
selection.  The  degree  of  reduction  in  error  rate  as  a  consequence  of 
special  training  is  yet  to  be  determined.  In  operational  settings,  this 
problem  of  error  rate  will  be  compounded  by  the  concomitant  screening, 
translation  (input  and  output),  and  error- correction  requirements  which 
also  will  beset  the  user.  Second,  the  time  results.  The  time  involved 
in  selecting  a  format,  considered  in  relation  to  the  message  load  to  be 
expected  in  peak  periods,  indicates  that  at  certain  times  the  staff 
action  officer  will  be  quickly  overloaded.  This  situation,  too,  will  be 
compounded  by  the  fact  that  for  some  messages  multiple  formats  will  be 
required,  increasing  selection  time.  Third,  the  indicated  confusibn 
between  class  of  message  and  format  selection.  Messages  in  certain 


classes  will  result  in  errors  more  often  than  others.  Overall,  the  most 
likely  error  is  that  data  will  be  sent  as  Relay  Messages  when,  in  fact, 
they  should  have  been  input  into  TOS  to  enrich  the  data  base. 

In  the  present  study,  concern  was  with  a  "simulated"  G3  staff  action 
officer  trying  to  match  an  incoming  message  with  one  of  44  possible 
formats  in  one  functional  area  (Friendly  Unit  Information).  When  the 
present  TOS  goes  into  the  field,  three  other  first  priority  functional 
areas  (Enemy  Situation,  Nuclear  Fire  Support,  and  Effects  of  Enemy 
Nuclear  Strikes)  will  be  added,  and  there  will  be  approximately  140 
formats  to  match  to  incoming  messages.  It  has  been  hypothesized  that 
the  total  number  of  different  formats  in  TOS  will  exceed  500  if  all 
planned  additional  functional  areas  are  introduced  into  the  system. 
Consequently,  transform  process  problems  should  be  addressed  now  before 
they  reach  major  proportions. 

One  task  that  should  be  undertaken  is  that  of  scrutinizing  existing 
formats  to  determine  if  the  absolute  number  can  be  reduced.  Considerable 
overlap  in  content  exists  among  the  present  formate.  Certain  formats 
could  conceivably  be  combined  because  of  this  communal ity.  Even  if  the 
present  "people  problem"  was  not  a  consideration,  the  sheer  cost  and 
logistic  problems  involved  in  printing,  transporting,  and  field  storage 
of  so  large  a  number  of  different  forms  dictate  a  critical  review  of 
requirements . 

Considering  the  normative  performance  data  developed  in  the  study, 
and  in  light  of  the  logistic  considerations,  the  question  of  need  for 
format  S6ilection  as  an  off-line  step  in  the  transform  process  is  raised. 
Viewed  simply  from  the  standpoint  of  data  through-put,  this  step  in  the 
operation  is  redundant.  The  staff  action  officer  fills  out  a  format 
worksheet.  He  then  hands  it  to  the  UIOD  operator.  The  UIOD  operator, 
in  turn,  calls  up  on  his  display  the  same  format  and  enters  the  identical 
information  into  the  system.  Toward  eliminating  this  redundancy  in  the 
operation  (which  inflates  system  process  time  and  introduces  anotner 
potential  source  of  system  error),  an  empirical  test  of  the  feasibility 
of  on-line  composition  should  be  undertaken.  Should  this  approach  prove 
feasible,  it  would  nor  only  improve  system  performance  but  would  also 
reduce  the  manpower  requirements  of  the  TOS. 

While  the  procedures  described  above  might  reduce  data  through-put 
time  as  well  as  eliminate  a  step  where  error  can  creep  into  the  system, 
there  remains  the  question  of  how  to  resolve  user  confusion  resulting 
from  irrelevant  correspondence  between  message  content  and  TOS  message 
category  titles.  As  a  short-term  approach,  training  which  focuses  on  the 
definitive  purpose  of  each  format  seems  reasonable.  The  efficacy  of  this 
approach,  however,  remains  to  be  demonstrated. 

This  aspect  of  the  transform  process  in  TOS  will  probably  be  with  us, 
in  one  guise  or  another,  for  some  time  to  come.  Studies  at  the  practical 
working  level  (for  example,  a  determination  of  the  possible  facili'  cing — 
or  debilitating — effects  of  on-line  composition  in  the  transform  operation) 


need  to  be  supplemented  by  studies  at  a  more  fundamental  level.  Many 
changes  between  the  "experimental"  and  the  ’'ultimate"  TOS  can  be  antici¬ 
pated.  Therefore,  in  order  to  recommend  what  should  be  done  in  the  future 
to  enhance  performance,  a  deeper  understanding  of  the  fundamental  process 
involved  is  required.  In  short,  the  research  should  involve  tasks  which 
are  analogous  to,  if  not  exactly  equivalent  to,  TOS  operations  as  pres¬ 
ently  envisioned.  Conceptualizing  the  task  in  the  abstract  will  permit 
free  manipulation  of  important  variables  in  a  way  that  is  precluded  in 
practical  experiments.  The  latter  normally  reduce  to  demonstrations  of 
the  efficiency  of  new  approaches  formulated  by  extrapolating  from  funda¬ 
mental  research.  In  this  case,  there  is  a  lack  of  appropriate  fundamental 
research  which  can  be  brought  to  bear  on  TOS-type  problems. 


Appetuix 


A.  Continuation  of  Staff  Action  Officer's  Translation 
Prncers  in  the  Transform  Operation 

figure  A-l .  Sample  of  a  completed  Unit  Status  Query 
worksheet 

B.  Instructions  for  Standard  and  Job-Aided  Group 


APPENDIX  A 

CONTINUATION  OF  STAFF  ACTION  OFFICER'S 
TRANSLATION  PROCESS  IN  THE  TRANSFORM  OPERATION 

Our  hypothetical  staff  action  officer  had  just  been  handed  the 
following  unclassified  message  from  the  GJ>  at  Seventh  Army  Main: 

"I  WANT  TO  KNOW  IMMEDIATELY  ALL  U.  S.  COMBAT  UNITS 
SUBORDINATE  TO  V  CORPS,  OF  DIVISION  ECHELON  OR  LESS, 

WHICH  HAVE  MORE  THAN  80  PERCENT  10E  PERSONNEL  AND 
MORE  THAN  70  PERCENT  COMBAT  EFFECTIVENESS .  ADDITIONALLY, 

THESE  UNITS  MUST  HAVE  AT  LEAST  700  PERSONNEL." 

Using  the  Unit  Status  Querji  Message  Worksheet,  the  staff  action 
officer  transformed  the  above  message  into  a  language  that  the  TOS  com¬ 
puter  understands  and  will  accept.  Shown  in  Figure  A-l  is  the  completed 
Unit  Status  Query  Worksheet  that  the  staff  action  officer  would  hand  to 
his  UlOn  (User  Input /Output  Device)  operator.  Let  us  examine  each  of  the 
entries  to  determine  how  he  transformed  the  basic  message  into  an  accept¬ 
able  "User/Computer  Language." 

PRECEDENCE:  A  reply  to  the  message  was  requested  "immediately,"  thus 
che  "I"  entry. 

HARDCOPY:  The  "Y"  indicates  that  the  Staff  Action  Officer  requested 

a  printed  copy  of  the  message  for  his  record  purposes. 

ORIGIN:  The  originator  of  the  message  is  the  G3,  Seventh  Army  Main. 

Therefore,  the  staff  action  officer's  originator  code  is 
"CMG3  • " 

SECURITY:  The  security  of  the  message  was  unclassified,  or  "UNCLAS." 

PERSONNEL:  The  number  of  personnel  stated  in  the  basic  message  was 

"at  least  700."  The  entry  for  this,  found  in  the  Rational 
Operator  (R-O)  list  of  his  glossary,  was  "NOLESS."  The 
number  (A-N)  700  is  entered  in  the  spaces  following  the 
slash. 

PERS-PCT:  The  message  specified  "more  than  80  percent  TOE  personnel." 

The  Relational  Operator  (R-O)  list  in  his  glossary  was  used 
to  supply  the  entry  "MORE."  The  number  "80"  is  the  percent 
of  TOE  entry  inserted  after  the  slash. 

CBT- EFFECT:  The  message  stated  "more  than  70  percent  combat  effectiveness. 

"MORE"  and  **7C"  were  entered  in  the  same  manner  as  above. 


ECKCT.ON: 


The  only  echelons  requested  for  the  output  message  were 
’'division  or  less."  The  two  entries  for  echelon,  therefore, 
were  "NGtfORE"  and  "DIV." 

CA13G0RY:  Th?  message  specified  "combat"  units;  therefore,  "CBT"  Was 

chosen  from  his  glossary. 

NATION:  Only  "US"  units  were  requested. 

SUBOR-TO:  The  message  indicated  that  only  units  r,enbc  -lit.  .ste  to  V  Corps 

were  to  be  reported.  The  entry  ’^-COUPS"  therefore  was  made. 

Thus,  the  original  free-2nglish  text  was  transformed  into  r*  computer- 
acceptaole  language.  As  noted  earlier,  the  language  required  is  cost 
precise,  e.g.,  the  SUBCFD-TC  entry,  ’^CORPS,"  must  he  written  exactly 
as  shown,  including  the  arabic  numeral  and  the  dash  (-),  or  the  computer 
will  reject  the  message. 
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f:'i 

s§ 


; -•  5 


IKSTOUCTI01TS  (Standard  Group) 

You  are  all  aware  of  the  numerous  formats  which  exist  for  inputting 
information  into  TOS.  Choosing  a  correct  format  for  a  message  must  be 
accomplished  quickly  and  correctly  for  TOS  to  have  a  real  benefit  to  the 
user.  We  have  asked  for  your  help  today  in  order  to  gather  information 
which  might  help  us  determine  what  types  of  problems  an  action  officer 
fright  have,  choosing  a  correct  format. 

We  have  prepared  a  set  of  canned  situation  descriptions  which  are 
similar  to  those  which  might  be  expected  in  the  G3  Friendly  Unit  Infor¬ 
mation  Area.  We  would  like  you  to  read  each  situation  description, 
choose  a  format  which  is  most  appropriate  for  TOS  to  handle  the  problem 
described,  and  write  that  format  code  in  the  space  provided  on  the  situa¬ 
tion  sheet.  Since  each  situation  represents  an  isolated  and  independent 
case,  it  should  not  be  considered  in  relation  to  ocher  situations. 

After  completing  your  answer,  read  the  next  situation  description 
and  do  not  refer  again  to  situations  previously  scored.  Since  the  number 
of  situations  to  be  scored  is  greater  chan  the  number  of  formats  to 
choose  from,  it  is  obvious  chat  some  formats  must  be  used  more  than  once. 
Also,  since  each  of  you  has  a  situation  description  set  which  was  randomly 
compiled.  It  may  be  that  some  of  you  will  find  that  some  message  formats 
need  not  be  used  at  all. 


In  addition  to  recording  ycur  answers  regarding  which  format  you 
selected  for  each  message,  we  want  to  find  out  how  long  it  took  you  to 
select  formats  for  all  of  the  messages  you  have.  This  will  be  done  by 
keeping  track  of  when  you  start  the  first  message  and  when  you  have 
selected  the  format  for  the  last  message.  Therefore,  you  must  let  us 
know  as  soon  as  you  finish.  Finally,  after  each  format  is  selected  you 
are  asked  to  rate  how  certain  you  are  that  you  have  selected  the  proper 
format. 

In  addition  to  the  set  of  situation  descriptions  which  you  will  be 
given,  you  now  have  a  list  of  message  formats  from  which  you  are  to  choose 
the  appropriate  formats  for  each  situation.  This  list  provides  the  three- 
character  format  codes  which  you  should  use  for  scoring  your  answers. 

Every  situation  in  the  set  given  you  has,  among  the  formats  listed,  one 
which  is  most  appropriate. 


You  are  all  aware  of  the  numerous  formats  which  exist  for  inputting 
information  into  TOS.  Choosing  a  correct  format  for  a  message  must  be 
accomplished  quickly  and  correctly  for  TOS  to  have  a  real  benefit  to  the 
user.  We  have  asked  for  your  help  today  in  order  to  gather  information 
which  might  help  us  determine  what  types  of  problems  an  action  officer 
might  have  choosing  a  correct  format. 


We  have  prepared  a  set  of  canned  situation  descriptions  which  are 
similar  to  those  which  might  be  expected  in  the  GJ  friendly  Unit  Informa¬ 
tion  Area.  We  would  like  you  to  read  each  situation  description,  choose 
a  format  which  is  most  appropriate  for  TOS  to  handle  the  problem  described, 
and  write  that  format  code  in  the  space  provided  on  the  situation  sheet. 
Since  each  situation  represents  an  isolated  and  independent  case,  it 
should  not  be  considered  in  relation  to  other  situations. 

After  completing  your  answer,  read  the  next  situation  description 
and  do  not  refer  again  to  situations  previously  scored.  Since  the  number 
of  situations  to  be  scored  is  greater  than  the  number  of  formats  to  choose 
from,  it  is  obvious  that  some  formats  must  be  used  more  than  once.  Also, 
since  each  of  you  has  a  situation  description  set  which  was  randomly 
compiled,  it  may  be  that  some  of  you  will  find  that  some  message  formats 
need  not  be  used  at  all. 

In  addition  to  recording  your  answers  regarding  which  format  you 
selected  for  each  message,  we  want  to  find  out  how  long  it  took  you  to 
select  formats  for  all  the  messages  you  tu.ve.  This  will  be  done  by 
keeping  track  of  when  you  start  to  format  the  first  message  and  when  you 
have  selected  the  format  for  the  last  message.  Therefore,  you  must  let 
us  know  as  soon  as  you  finish.  Finally,  after  each  format  is  selected, 
you  are  asked  to  rate  how  certain  you  are  chat  you  have  selected  the 
proper  format. 

In  addition  to  the  set  of  situation  descriptions  which  you  will  be 
given,  you  now  have  a  decision  chart  which  will  assist  you  to  find  the 
correct  format.  The  chart  is  arranged  to  indicate  the  correct  format 
code,  given  that  you  know  the  type  of  message  yea  need  to  send  and  the 
sabject  content  of  the  message. 

After  reading  a  situation  description,  begin  at  the  top  of  the  chart 
and  answer  the  question,  "What  do  you  want  to  do?"  Then  look  at  the 
column  which  contains  your  answer  to  this  question,  e.g.,  modify  the  data 
base.  Two  columns  ask  che  additional  question,  "How?"  You  could  query 
the  data  base  more  than  once  by  establishing  an  SRI  or  changing  an  exist¬ 
ing  one.  Also,  you  can  modify  the  data  base  by  adding  data,  deleting  data, 
or  changing  data. 

Once  you  have  established  the  correct  column,  find  the  row  which 
indicates  the  subject  of  the  message  you  wish  to  send.  The  intersection 
of  the  row  and  column  gives  the  correct  format  code. 
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1*.  ABSTRACT 


Under^dSESSir^s  COMMAND  SVCTEMS  program,  specific  aspects  of  information  presen 
tation,  processing,  and  utilization  in  command  and  control  systems  are  examined  with  re 
spect  to  the  requirements  for  and  capabilities  of  handling  and  presenting  information  t 
human  users  of  the  system.  The  present  study  was  conducted rby— BESRL*s  Command  Systems 
Field-Branch  in  Europe  to  examine  systematically  human  factors  problems  related  to  erro 
rate,  processing  time.  And  confidence  in  message  format  selection.  A  second  objective 
was  to  evaluate  a  newly  devised  job  aid  for  use  in  selecting  appropriate  formats. ^The 
experimental  TOS,  being  developed  and  evaluated  as  part  of  an  Army-wide  program  (*3SAF) 
utilises  over  40  different  message  formats  from  among  which  G3  staff  action  officers 
have  to  determine  the  appropriate  format  or  formats  to  use  with  each  set  of  incoming 
data.  Of  concern  in  this  study  were  the  human  factors  problems  involved  in  transformin 
free-Rnglish  G3  message  texts  into  rigidly  defined  format  in  preparation  for  user  inter 
action  with  the  TOS .  In  the  experiment,  47  simple  messages  were  given  to  14  individual 
familiar  with  TOS  or  G5  operations,  or  both.  Half  the  men  (job-aid  group)  used  an  aid 
devised  by  the  BE1SRL  investigators,  and  half  (standard  group)  used  a  "menu"  type  listin 
of  available  formats  in  performing  the  required  task  of  selecting  an  appropriate  format 
for  eAch  message.  Time  taken  to  complete  the  task  and  also  degree  of  confidence  in  the 
selection  made  were  recorded.  Findings  showed  a  mean  time  of  approximately  50  seconds 
for  message  reading  and  format  selection.  Average  error  rate  in  format  selection  was 
22$ .  No  significant  difference  was  obtained  in  performance  with  the  job  aid  over  that 
with  the  "menu"  type  listing. '  Messages  of  different  types  and  subject  matter  differed 
in  error  rate  of  format  selection.  Mean  expressed  certitude  of  correctness  in  format 
selection  was  almost  th6  same  for  both  groups.  Time  and  error  rate  data,  together  with 
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15 .  ABSTRACT  -  Continued 


other  factors  such  as  screening  for  applicability  to  TOS,  correction  of 
errors,  and  translating  more  complex  tests  led  to  the  conclusion  that 
the  transformation  process  is  potentially  a  major  problem  in  the  experi¬ 
mental  TOS.  Results  provided  baseline  data  for  comparative  study  of 
future  performance  and  suggested  some  approaches  to  training  and  alterna¬ 
tive  methods  for  the  transform  process. 
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